<!DOCTYPE html>
<html lang="en">
<head>
    <!-- hexo-inject:begin --><!-- hexo-inject:end --><meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="google-site-verification" content="4oDu9FBIEUQY3tUyERR6kppEHHZi9LdJmulKg0v0wuU" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="Fainle的博客">
    <meta name="keyword"  content="github">
    <link rel="shortcut icon" href="/img/favicon.ico">

    <title>
        
        六：梯度下降（Gradient Descent） - Fainle的博客 | Fainle&#39;s Blog
        
    </title>

    <!-- Custom CSS -->
    <link rel="stylesheet" href="/css/aircloud.css">
    <link rel="stylesheet" href="/css/gitment.css">
    <!--<link rel="stylesheet" href="https://imsun.github.io/gitment/style/default.css">-->
    <link href="//at.alicdn.com/t/font_620856_pl6z7sid89qkt9.css" rel="stylesheet" type="text/css">
    <!-- ga & ba script hoook -->
    <script></script><!-- hexo-inject:begin --><!-- hexo-inject:end -->
</head>

<body>

<!-- hexo-inject:begin --><!-- hexo-inject:end --><div class="site-nav-toggle" id="site-nav-toggle">
    <button>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
    </button>
</div>

<div class="index-about">
    <i> 人生苦短，多思考，少编程 </i>
</div>

<div class="index-container">
    
    <div class="index-left">
        
<div class="nav" id="nav">
    <div class="avatar-name">
        <div class="avatar ">
            <img src="/img/avatar.jpg" />
        </div>
        <div class="name">
            <i>Fainle</i>
        </div>
    </div>
    <div class="contents" id="nav-content">
        <ul>
            <li >
                <a href="/">
                    <i class="iconfont icon-shouye1"></i>
                    <span>主页</span>
                </a>
            </li>
            <li >
                <a href="/tags">
                    <i class="iconfont icon-biaoqian1"></i>
                    <span>标签</span>
                </a>
            </li>
            <li >
                <a href="/archive">
                    <i class="iconfont icon-guidang2"></i>
                    <span>存档</span>
                </a>
            </li>
            <li >
                <a href="/about/">
                    <i class="iconfont icon-guanyu2"></i>
                    <span>关于</span>
                </a>
            </li>
            
            <li>
                <a id="search">
                    <i class="iconfont icon-sousuo1"></i>
                    <span>搜索</span>
                </a>
            </li>
            
        </ul>
    </div>
    
        <div id="toc" class="toc-article">
    <ol class="toc"><li class="toc-item toc-level-3"><a class="toc-link" href="#1-概述"><span class="toc-text">1 . 概述</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-SSE"><span class="toc-text">2 . SSE</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#3-代码实现"><span class="toc-text">3 . 代码实现</span></a></li></ol>
</div>
    
</div>


<div class="search-field" id="search-field">
    <div class="search-container">
        <div class="search-input">
            <span id="esc-search"> <i class="icon-fanhui iconfont"></i></span>
            <input id="search-input"/>
            <span id="begin-search">搜索</span>
        </div>
        <div class="search-result-container" id="search-result-container">

        </div>
    </div>
</div>

        <div class="index-about-mobile">
            <i> 人生苦短，多思考，少编程 </i>
        </div>
    </div>
    
    <div class="index-middle">
        <!-- Main Content -->
        


<div class="post-container">
    <div class="post-title">
        六：梯度下降（Gradient Descent）
    </div>

    <div class="post-meta">
        <span class="attr">发布于：<span>2020-07-28 17:02</span></span>
        
        <span class="attr">标签：/
        
        <a class="tag" href="/tags/#深度学习笔记" title="深度学习笔记">深度学习笔记</a>
        <span>/</span>
        
        
        </span>
        <span class="attr">访问：<span id="busuanzi_value_page_pv"></span>
</span>
</span>
    </div>
    <div class="post-content ">
        <h3 id="1-概述"><a href="#1-概述" class="headerlink" title="1 . 概述"></a>1 . 概述</h3><p>梯度下降法（英语：Gradient descent）是一个一阶最优化算法，通常也称为最陡下降法，主要使用梯度下降法找到一个函数的局部极小值，必须向函数上当前点对应梯度（或者是近似梯度）的反方向的规定步长（学习率）距离点进行迭代搜索。</p>
<h3 id="2-SSE"><a href="#2-SSE" class="headerlink" title="2 . SSE"></a>2 . SSE</h3><script type="math/tex; mode=display">E = \frac{1}{2}\sum_\mu(y^{\mu}-\hat{y}^{\mu})^2</script><script type="math/tex; mode=display">\hat{y} = \sigma(\sum_{i}w_{i}x_{i}^{u})</script><script type="math/tex; mode=display">E = \frac{1}{2}\sum_\mu(y^{\mu}- \sigma(\sum_{i}w_{i}x_{i}^{u}) )^2</script><h3 id="3-代码实现"><a href="#3-代码实现" class="headerlink" title="3 . 代码实现"></a>3 . 代码实现</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">plot_points</span><span class="params">(X, y)</span>:</span></span><br><span class="line">    admitted = X[np.argwhere(y==<span class="number">1</span>)]</span><br><span class="line">    rejected = X[np.argwhere(y==<span class="number">0</span>)]</span><br><span class="line">    plt.scatter([s[<span class="number">0</span>][<span class="number">0</span>] <span class="keyword">for</span> s <span class="keyword">in</span> rejected], [s[<span class="number">0</span>][<span class="number">1</span>] <span class="keyword">for</span> s <span class="keyword">in</span> rejected], s = <span class="number">25</span>, color = <span class="string">'blue'</span>, edgecolor = <span class="string">'k'</span>)</span><br><span class="line">    plt.scatter([s[<span class="number">0</span>][<span class="number">0</span>] <span class="keyword">for</span> s <span class="keyword">in</span> admitted], [s[<span class="number">0</span>][<span class="number">1</span>] <span class="keyword">for</span> s <span class="keyword">in</span> admitted], s = <span class="number">25</span>, color = <span class="string">'red'</span>, edgecolor = <span class="string">'k'</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">display</span><span class="params">(m, b, color=<span class="string">'g--'</span>)</span>:</span></span><br><span class="line">    plt.xlim(<span class="number">-0.05</span>,<span class="number">1.05</span>)</span><br><span class="line">    plt.ylim(<span class="number">-0.05</span>,<span class="number">1.05</span>)</span><br><span class="line">    x = np.arange(<span class="number">-10</span>, <span class="number">10</span>, <span class="number">0.1</span>)</span><br><span class="line">    plt.plot(x, m*x+b, color)</span><br><span class="line">    </span><br><span class="line">    </span><br><span class="line">data = pd.read_csv(<span class="string">'data.csv'</span>, header=<span class="keyword">None</span>)</span><br><span class="line">x = np.array(data[[<span class="number">0</span>,<span class="number">1</span>]])</span><br><span class="line">y = np.array(data[<span class="number">2</span>])</span><br><span class="line">plot_points(x,y)</span><br><span class="line">plt.show()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">sigmoid</span><span class="params">(x)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    sigmoid 计算</span></span><br><span class="line"><span class="string">    s(x) = 1/ (1 + e^-x)</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="keyword">return</span> <span class="number">1</span> / (<span class="number">1</span> + np.exp(-x))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">output_formula</span><span class="params">(features, weights, bias)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    拟合函数</span></span><br><span class="line"><span class="string">    x : 特征</span></span><br><span class="line"><span class="string">    w : 权重</span></span><br><span class="line"><span class="string">    b : 偏移</span></span><br><span class="line"><span class="string">    y = s(Wx) + b</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="keyword">return</span> sigmoid(np.dot(features, weights) + bias)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">error_formula</span><span class="params">(y, output)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    错误率计算公式</span></span><br><span class="line"><span class="string">    y : 预测值</span></span><br><span class="line"><span class="string">    f : output_formula</span></span><br><span class="line"><span class="string">    error = -ylog(f)-(1-y)log(1-f)</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="keyword">return</span> np.dot(-y, np.log(output)) - np.dot((<span class="number">1</span> - y), np.log(<span class="number">1</span>-output))</span><br><span class="line"><span class="comment">#     return -y * np.log(f) - (1 - y) * np.log(1 - f)</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">update_weights</span><span class="params">(x, y, weights, bias, learnrate=<span class="number">0.1</span>)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    更新权重</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    output = output_formula(x, weights, bias)</span><br><span class="line">    d_error = -(y - output)</span><br><span class="line">    weights -= learnrate * d_error * x</span><br><span class="line">    bias -= learnrate * d_error</span><br><span class="line">    <span class="keyword">return</span> weights, bias</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">train</span><span class="params">(features, targets, epochs, learnrate, graph_lines=False)</span>:</span></span><br><span class="line">    </span><br><span class="line">    errors = []</span><br><span class="line">    n_records, n_features = features.shape</span><br><span class="line">    last_loss = <span class="keyword">None</span></span><br><span class="line">    weights = np.random.normal(scale=<span class="number">1</span> / n_features**<span class="number">.5</span>, size=n_features)</span><br><span class="line">    bias = <span class="number">0</span></span><br><span class="line">    <span class="keyword">for</span> e <span class="keyword">in</span> range(epochs):</span><br><span class="line">        del_w = np.zeros(weights.shape)</span><br><span class="line">        <span class="keyword">for</span> x, y <span class="keyword">in</span> zip(features, targets):</span><br><span class="line">            output = output_formula(x, weights, bias)</span><br><span class="line">            error = error_formula(y, output)</span><br><span class="line">            weights, bias = update_weights(x, y, weights, bias, learnrate)</span><br><span class="line">        </span><br><span class="line">        <span class="comment"># Printing out the log-loss error on the training set</span></span><br><span class="line">        out = output_formula(features, weights, bias)</span><br><span class="line">        loss = np.mean(error_formula(targets, out))</span><br><span class="line">        errors.append(loss)</span><br><span class="line">        <span class="keyword">if</span> e % (epochs / <span class="number">10</span>) == <span class="number">0</span>:</span><br><span class="line">            print(<span class="string">"\n========== Epoch"</span>, e,<span class="string">"=========="</span>)</span><br><span class="line">            <span class="keyword">if</span> last_loss <span class="keyword">and</span> last_loss &lt; loss:</span><br><span class="line">                print(<span class="string">"Train loss: "</span>, loss, <span class="string">"  WARNING - Loss Increasing"</span>)</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                print(<span class="string">"Train loss: "</span>, loss)</span><br><span class="line">            last_loss = loss</span><br><span class="line">            predictions = out &gt; <span class="number">0.5</span></span><br><span class="line">            accuracy = np.mean(predictions == targets)</span><br><span class="line">            print(<span class="string">"Accuracy: "</span>, accuracy)</span><br><span class="line">        <span class="keyword">if</span> graph_lines <span class="keyword">and</span> e % (epochs / <span class="number">100</span>) == <span class="number">0</span>:</span><br><span class="line">            display(-weights[<span class="number">0</span>]/weights[<span class="number">1</span>], -bias/weights[<span class="number">1</span>])</span><br><span class="line">            </span><br><span class="line"></span><br><span class="line">    <span class="comment"># Plotting the solution boundary</span></span><br><span class="line">    plt.title(<span class="string">"Solution boundary"</span>)</span><br><span class="line">    display(-weights[<span class="number">0</span>]/weights[<span class="number">1</span>], -bias/weights[<span class="number">1</span>], <span class="string">'black'</span>)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># Plotting the data</span></span><br><span class="line">    plot_points(features, targets)</span><br><span class="line">    plt.show()</span><br><span class="line"></span><br><span class="line">    <span class="comment"># Plotting the error</span></span><br><span class="line">    plt.title(<span class="string">"Error Plot"</span>)</span><br><span class="line">    plt.xlabel(<span class="string">'Number of epochs'</span>)</span><br><span class="line">    plt.ylabel(<span class="string">'Error'</span>)</span><br><span class="line">    plt.plot(errors)</span><br><span class="line">    plt.show()</span><br></pre></td></tr></table></figure>
        
        <br />
        <div id="comment-container">
        </div>
        <div id="disqus_thread"></div>

        <div id="lv-container">
        </div>

    </div>
</div>

    </div>
</div>


<footer class="footer">
    <ul class="list-inline text-center">
        
        

        

        

        
        <li>
            <a target="_blank"  href="https://github.com/Fainle">
                            <span class="fa-stack fa-lg">
                                <i class="iconfont icon-github"></i>
                            </span>
            </a>
        </li>
        

        
        <li>
            <a target="_blank"  href="https://www.linkedin.com/in/Fainle">
                            <span class="fa-stack fa-lg">
                                <i class="iconfont icon-linkedin"></i>
                            </span>
            </a>
        </li>
        

    </ul>
    
    <p>
        <span>/</span>
        
        <span><a href="https://fainle.github.io">fainle</a></span>
        <span>/</span>
        
    </p>
    
    <p>
        <span id="busuanzi_container_site_pv">
            <span id="busuanzi_value_site_pv"></span>PV
        </span>
        <span id="busuanzi_container_site_uv">
            <span id="busuanzi_value_site_uv"></span>UV
        </span>
        Created By <a href="https://hexo.io/">Hexo</a>  Theme <a href="https://github.com/aircloud/hexo-theme-aircloud">AirCloud</a></p>
</footer><!-- hexo-inject:begin --><!-- hexo-inject:end -->




</body>
<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>

<script>
    // We expose some of the variables needed by the front end
    window.hexo_search_path = "search.json"
    window.hexo_root = "/"
    window.isPost = true
</script>

<script src="https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js"></script>
<script src="/js/index.js"></script>
<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>


    <script>
        /**
         *  RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
         *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables
        */
        if( '' || '')
        var disqus_config = function () {
            this.page.url = '';  // Replace PAGE_URL with your page's canonical URL variable
            this.page.identifier = ''; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
        };

        (function() { // DON'T EDIT BELOW THIS LINE
            var d = document, s = d.createElement('script');
            s.src = 'https://fainle.disqus.com/embed.js';
            s.setAttribute('data-timestamp', +new Date());
            (d.head || d.body).appendChild(s);
        })();
    </script>



</html>
